Duplication in production code leads to errors, but certain duplication in test code causes problems too.
When test after test has the same structure, the part that varies (aka the interesting part) gets buried in the part that stays the same. This makes it hard to be sure you’ve covered all the right cases, and makes it hard to recognize when there are important but subtle variations between the tests.
Parameterized tests address this: extract the varying data into parameters, and pass them into a test method.
Types of Parameters
Most tests have three types of values: inputs, outputs (expectations), and metadata.
Inputs: Data used to configure objects, or passed as input to their methods.
Outputs (Expectations): Expected results of calling the functions or methods.
Metadata: Data about the test, typically including a description or message, and possibly including location data such as the line number.
If you omit inputs: You’re either testing a constant (which often isn’t worth the effort), or you’re testing something where the test isn’t controlling the situation. (This is unusual but not impossible: somebody has to test the random number generator.)
If you omit expected outputs: This suggests the output isn’t expected to vary. For example, you may be testing that exceptions are consistently thrown. Or, you may be writing a property-based test, checking that certain relationships hold for arbitrary inputs. For example, we can test arithmetic-like operators for associativity: for any a, b, and c, check that op(op(a,b),c) = op(a, op(b,c))
If you omit metadata: Your test still works, but it can be harder to interpret it – either to understand what you’ve covered, or to report it conveniently when it fails.
Automated and Manual Support
If you’re using JUnit, NUnit, or Cucumber, the standard package supports parameterized tests. Typically you can either define the test values inline, or use a separate function that provides a stream of values.
(I won’t address this case further.)
If your test framework doesn’t support parameterized tests (e.g., XCTest for Swift), you can simulate this by putting the test data in an array or other data structure, and looping through that to call a (parameterized) test method. Most xUnit frameworks stop testing when the first assert fails, but you can either live with this or take extra trouble to work around it.
A Swift Solution
I have a fairly simple approach to running parameterized tests in Swift. I can’t offer this as a “best possible” solution, but it meets a lot of what I want:
- Test data is centralized and visible
- All errors are reported in a single run (barring crashes) [provided by XCTest]
- It reports the line number where the data is from
You may find the style a bit terse, but I’m trying to make the mechanism fade so the data shines through.
This approach uses a struct (record type) that handles test parameters. It looks like this:
struct P<Input, Output> {
var input: Input
var output : Output
var message: String
var line: Int
init(input: Input, output: Output,
_ message: String = "", _ line: Int = #line) {
self.input = input
self.output = output
self.message = message
self.line = line
}
func msg() -> String {
return "Line \(line): \(message)"
}
}
The message is optional, as is the line number. Because it uses #line, the line number is filled in with the line number of the call site.
For test arguments, the generic parameters let you pass either single types or tuples. A tuple lets you mix types, e.g., (1, “str”, 3.14).
A test looks something like this:
[
// test data using P()
].forEach { p in
// test body here; access values through p
}
Example – Code Under Test
Here’s a class with a few defects.
class Demo {
func stringOfSum(_ a: Int, _ b: Int) -> String {
// Defect 1: this "if" check shouldn't be here
if (a + b) <= 0 {
return "??"
}
return String(a + b)
}
func xPrefix(_ s: String) throws -> String {
// Defect 2: should throw if input is all "0"
// Defect 3: should not throw if input is empty
if s.count == 0 { throw FilterError.invalidState }
return "x\(s)"
}
}
Example – Parameterized Unit Tests
Here’s a test class that uses P. It demonstrates passing and failing tests, and expected and unexpected exceptions.
class DemoFailingTests: XCTestCase {
func testStringOfSum() {
[
P(input: (-1, 1), output: "0", "zero"),
P(input: (3, 0), output: "3", "one-digit"),
P(input: (-2, 1), output: "-1", "negative")
].forEach { p in
let my = Demo()
let actual = my.stringOfSum(p.input.0, p.input.1)
XCTAssertEqual(p.output, actual, p.msg())
}
}
func testPrefix() throws {
try [
P(input: "3", output: "3", "wrong expected output"),
P(input: "y", output: "xy", "prepends x"),
P(input: "", output: "x", "empty should be ok")
].forEach { p in
XCTAssertEqual(
p.output, try Demo().xPrefix(p.input), p.msg())
}
}
func testPrefixThrows() throws {
try [
P(input: "0", output: (), "expected throw"),
P(input: "000", output: (), "expected throw"),
].forEach {p in
XCTAssertThrowsError(try Demo().xPrefix(p.input), p.msg())
}
}
}
I’ve coded the tests inline, but you could replace this with a method reference if you wanted more separation.
Example – Test Output
Here’s what the test looks like after it’s run (clicking on the red X). It tells all failures and where they’re located.
Conclusion
If you’re not using Swift, I still hope you’ll use the idea of input, output, and metadata parameters when you parameterize tests.
If you are using Swift, give this solution a try and let me know what you think.